OOCUMEHZ &ESUNE 



ED 186 6B(( 



CE 025 246 



AUTHOR 
TITLE 

INSIIIUtlON 

5P0NS AGENCY 

PUB DATS 

GRANT 

NOIE 



EDES .PRiCE • 
DiiSCfllPIOaS 



fCnapp, Joan £• 

Assessing X^ianscer Skills, 

Ohio State UniV., Columbus. National Center for 

Research in Vocational Education. 

National last, of Education (OHEW) , Washington, 

D.C, 

7y 

OB-NX£-G-7d-3211 

25p,; For related documents see ID 1 39 ?:D 1^6 

420, ED 146 i45ti, hU 174 809, CE 024 335-B36, CE 025 
246-^47, and CE 025 306. 

MfOI/PCOl Plus Postage. 

Educational Practices: Essay Tests; Hiaher Education; 
Individual Testing; Interviews; Job Skills; Learning 
Theories; Mastery Learning; Objective Tests; 
Performance Tests; Secondary Education; Self 
Evaluation (Individuals) ; Simulatioiu Skill 
Development; *SieilJu3; State of the ifrt Eeviews; 
^ .esting; ^Transfer ol Training 



ABSTRACT , , 

following a summary of what is presently known aoout 
transfer skills, the options available for assessment of transfer 
sk-ils are identified and problems inherent in that task examined. 
Transfer skills (the ability or capacity of the learner or worker tc 
transfer skills or knowledge from prior experiences to new onesi are 
discussed using the work of a naibet of. contemporary theorists and 
noting points of general agreement. Types of educational measurements 
that can oe applied to the assessment of transfer of skills are 
classified as either traditional or noi;traditional methods. 
Traditional methods examined are paper and pencil, multiple choice, 
and objective tests. Nontradit ional methods analyzed are interviews 
"and oral examinations; simulations; essays; performance tests; and 
self-assessment devices. -Examples of each, their use as well as 
advantages and disadvantages, are given. It is noted that no specific 
tests or procedures couli oe recoiuaended as end-alls; rather, they 
are'presented as tools to help determine if transfer has occurred. A 
bibliography of selected documents describing nontraditional 
assessment methods is includea a* an appendix. (MEKJ 
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FOREWORD 



Recent surveys indicate that job change, either by choice or by necessity, has become a fact 
of life for a substantii proportion of the adult population, in order to better serve their students 
and employees, educwon and work programs need a thorough understanding of the factors which 
improve the ability of individuals to adapt to such change. In an effort to provide this understanding, 
the National Center for Research in VocatioRal Education, under the sponsorship of the National 
Institute of Education, has been involved in the delineation of the factors which make an individual 
occupationatly adaptable. 

One of the concepts connected with occupational adaptability is that of transfer skills. These 
are skills which enable an, individual to take competencies, attitudes, and knowledge gained in a 
work or school setting and to apply them to tasks confronted in a new work environment. Since 
Individuals may not be able to recognize their own transfer skills, methods for the assessment of 
these skills are vitally important. This report reviews the options available for the assessment of 
transfer skills and identifies the problems inherent in this difficult task. 

The National Center expresses its appreciation to Joan E. Knapp for her scholarly efforts in 
the preparation of the report. We are especially indebted to Ann Withorn and Gary Woditsch for 
their critical review of the manuscript prior to final revision and publication. We are also grateful 
to Robert Abram, project coordinator, for his role in the development of the report; to Robert 
Stump, project monitor from the National Institute of Education, for his interest and helpful advice 
throughout the development of this report; to James Cellini for his careful editing of the final 
report; and tp John Crystal, Ruth Nickse, and Decker Walker for their invaluable aid as consultants 
to the Transfjerable Skills program. This report is oart of the Transferable Skills program directed 
by William Ashley. 



Robert E. Taylor 
Executive Director 
The National Center for Research 
in Vocational Education 



PREFACE 

When I was approached with the task of writing a paper on assessment for the National 
Center for Research in Vocational Education, I accepted enthusiastically for several reasons: 

(1 ) The study of measurement and assessment of prior learning from experience has taken up a 
significant portion of my professional worK. Thus, I thought it would give me an opportunity to 
apply various assessment concepts to an aspect of learning that was new to me— transfer skills; 

(2) I hoped I could produce a useful handbook for practitioners to use in educational assessment 
situations; (3) Finally, because of past work in assessment, I smugly thought it would be a relatively 
simple assignment, which added even more to my enthusiasm. 

In practice, the ^ask turned out to be, in its early stages, a puzzlement, and in its later stages 
a challenge. At the end, i realized that my goals for a handbook were premature and somewhat 
ambitious. I learned that the term "transfer skills" was not well defined in educational psychology 
and measurement. Semantic confusions in the social sciences are not unusual. But the aura of 
uncertainty about "transfer skills" was confirmbd when I searched various literature data bases and 
found few references under "transfer skills," wiiereas, for example, the term "intelligence" gave me 
more than I cared to examine. 

Further investigations convinced me that there was a significant body of research in "transfer 
of training" which peaked in the early sixties. I feel that this research is soon to be revived. 
Researchers and educators do believe that transfer ability or the capacity for transfer can be 
defined and is a construct subject to research and, more importantly, is teachable. I feel the new 
work will veer away from the methodology of the past that focused on highly controlled experi- 
mentation using nonsense syllables and simple motor tasks to studies that are more naturalistic and 
involve the training and eliciting of complex transfer competencies. 

This paper represents an attempt to tell briefly what is known about transfer and to generally 
touch upon the usefulness of various assessment procedures to the transfer process. 
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ASSESSING TRANSFER SKILLS 



What are Transfer Skills? 

The terms transfer skills or transfer abilities will be used frequently throughout this paper- 
From the outset, a distinction needs to be made between transfer skills and transferable skills. 
Transferable skills are those abilities and competencies that can be appUed in a variety of work 
settings as well as in almost every aspect of life. Although there are a number of taxonomies that- 
list and describe functional skills necessary for occupational and life adaptability, there is no wide- 
spread agreement on one particular core set of transferable skills^ Nevertheless, through basic 
research in learning psychology it is known that some skills are more transferable than others and 
that the ease and effectiveness of transfer depends on a nuniber of factors. ^ 

One of the major factors that influences the transfer of skills and training is the extent and 
level of transfer skills possessed by an individual. What is the ability or capacity of the learner or - 
worker to transfer one or several skills from one setting to another? For example, a worker has the 
ability to estimate time, weight, and distance on an assembly line (transferable skills). Can these 
skills be transferred to a sales job where these or comparable skills are needed to estimate the costs 
of a product for a potential customer or buyer? Can these skills be transferred to a hobby like 
furniture-building? What are the skills or abilities that lead to transfer? 

Although much research has been conducted about transfer of learning, a precise list of the 
cognitive factors that aid skill transfer does not exist. However, it is thought by some psychologists 
that cognitive abilities or factors such as the following could be the catalysts without which the 
process of transfer could not occur. ^ 

1. Cue recognition: ability to identify and recognize significant signals, signs, and inputs 
through the senses and relate them to relevant factors and/or situational variables. 

2. Discrimination: ability to discern significant differences in object, events, and ideas 
and sort or group according to difference. 

3. Association: ability to recognize significant similarities between objects, events, and 
ideas and to order, organize, or group according to similarity. 

4. Reasoning by analogy: ability to develop and apply a rational line of reasoning 
drawing upon comparisons of analogous situations, events, and characteristics. 

5. Rule or theory application: ^ability to interpret and apply a rule, strategy guideline, 
or theory in varying contexts and circumstances. 



See the following for two different sets of trar sferable skills 

Wiant, A. A. Transferable Skills: The employer's viewpoint. Columbus, OH: The Ohio State University, The National Center for 
Research m Vocaxional Education, 1977. (Info. Series No. 126) 

Kawj!:;. 1 J. J . and Smith, A. D. Generic skills: Handbook of occupational information. Prince Albert, SK: Canada Manpower 
and Immigration Department, Trainmg Research and Development Station, 1975. 
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6. Information access: skill in finding and interpreting information from a variety of 
extant sources. 

7. Information production: skill in producing and recording information in a variety of 
formats and styles for the purpose of transfer and storage. 

8 Synthesis: ability to compile and condense informatipn, data, or concepts into 

concise statements, findings, or propositions that express new or different relationships. 

9. Decision making: ability to establish a rational criterion for selection and apply it to 
* two or more alternative courses of action to choose one. 

10. Solution generation: ability to generate a variety of alternative solutions or probable 
solutions for a given problem. 

1 1 Short-term planning: ability to organize a series of immediate events, tasks, or activities 
into a sequence and resources to accomplish a single goal in a short time frame (hours/ 
days). 

1 2 Long-term planning: ability to organize a series of broad functions, processes, and/or 
activities and resources in order to accomplish one or more goals over a long time frame 
(weeks/months). 

13. Value analysis: ability to rationally compare/contrast aiternativepayoffs or outcomes 
and determine probable utility of each one in comparison to all others. 

Another way of looking at transfer skills and abilities does not compartmentalize them into a 
lona list of discrete cognitive factors but looks at them in terms of patterns of behavior that are 
likely to lead to successful or unsuccessful transfer. The contexts in which these mental operations 
are likely to succeed are equally important. One model that has been proposed, states that there are 
aeneric skills that are basically a small set of mental operations and appear again and again as com- 
ponents of successful learning behavior.^ Further, using a biological analogy, these are not available 
for direct inspection (genotypic); we only know these skills when they are actually employed (pheno- 
typic), and lead to successful leaming transfer. Those behaviors sufficient for smooth conceptual 
transfer might be: 

1 . Selective attention: ability to control the class of stimuli which receive conscious focus. 

2. Sustained analysis: a capacity to probe a complex situation until all its components 
are identified. 

3. Analogizing: a capacity to test known relationships for similarity with those potential 
to a new situation. 

4. Suspension of closure: prioritizing (synthesizing) factors before shaping solution. 

5. Autocensorship: testing a solution covertly, before*affirmation. 



i 



2 Wod.tsch G. A. De^elop.ng genenc skills: A model for competency based general edacat.on. Bowlmg Green. OH: Bowlinq Green 
St.ite University. CUE Pfoiect, 1977 (Occasional Paper Series No. 3). 
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Conversely, research shows forms of behavior that accompany learning failure are also readily ^ 
observed: • . " 

1. Learner exhibits random attention. 
<^ 2. Learner scans compulsively and haphazardly. . ' ^ 

3. Learner fails to test prior knowledge against. potential relationships in a problem. 

4. Learner guesses chronically. 

« 

5. ^ Learner fails to check solution to problem. . . * 

The Woditsch model also is in agreement with the large body of transfer of traihing research 
that attempts to investigate the conceptual and motivational as well as cognitive factors that strongly 
influence the transfer of skills.^ 

A great deal of this research focuses on fairly trivial and narrow (not generic) skills because it 
was thought that controlled experiments would lead to a clearer understanding of the phenomenon. 
Much was learned from these investigations. The major finding was that the transfer process was 
extraordinarily complex. This realization jed to a variety of theories that attempt to explain what 
happens when learning is transferred * No matter what theory is espoused there is some agreement 
about the following: 

1. All skills are potentially transferable. . ^ 

2. Skills must be developed to high levels of mastery if positive transfer is to occur. 

3. Heightened transfer occurs wt^en the content, context, and process of learning are 
similar from one situation to another. When any of these vary, transfer is diminished. 

4. Those cognitive operations that aid in transfer are closely related to the psychological 
constructs labeled "intelligence" or "problem-solving ability." 

5. Relaying performance information (feedback) to the learner maximizes skill transfer. 
The more similar the feedback from one situation to the other, the greater the potential 
for transfer. 

6. Once a skill is transferred or a problem is solved there is confidence it can be done again. 

7. Transfer is accomplished more easily if the learner can discover the relationships between 
the skill(s) and situation(s) for him or herself. . 

.8. Transfer skills are perishable. The transfer facility and its retention depends on practice, 
attitude, and the biological age of the learner. 

9. Transfer is aided by making the goals of training explicit and exposing the learner to the 
variety of situations in which he or she may exercise skills. 



for sumiryiries of this research see: 

Altman, J. W. Transferability of vocational skills: Review of literature and research. Columbus. QH. The Ohio State University, 
The National Center for Research in Vocational Education, 1976i Info. Series No. 103. 

Stolurow, L. M. Psychological and educational factors in transfer of tra.ning. Section 1 Final Report. Urbana, IL: University ot 
Illinois. Bureau of Educational Research, July 1966. 

^Royer, J. M. Theories of transfer of learning. Educational Psychologist, 1979, M, 53-69. 
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10. The extent to which a skill can be transferred depends on: » ) 

a. type of skill and thi onditions of practice; * 
■ b. anxiety level^of the learner; 

c. sex and minority status of the leerner; / 
. d. reward system . - ; 

11. The facility for transfer can be taught, yet our educational system frequency does not 
ttach these skid* Many programs j)pt. only do not teach for tran§fer,;|k)ut existing 
teaching and assessment nr>ethod's of tet^,fe^tard transfer. ^ 

Another way of describing the process of transfer is td say ^hat tr^n:feraljilitY is really a 
complex master competence that results from highly integrated cognitive-factors. This competence 
is really what many call problem-solving ability, adaptability, flexijiUity (as opposed to rigidity), or 
Insight, which is the rapid solution of problems resulting from" exjjfentrve^awtice. 

Clearly, much of the teaching and testing^ring the education of a typical individual occurs 
in an academic, nonrealistic setting. Knowledge acquisition is emphasized over application and 
analysis. If problem-solving is emphasized, it is assumed that there is one solution t© the problem 
and one route to get that solution. Unfortunate^^, skill trar^sfer on the job and ia life is quite 
different. A clear example of the lirpits of schooling is coiptained in a story about the Kpelje, an 
articulate, unschooled people of. Liberia. The Kpelle do poorly on tests and problems that seem 
easy to their schooled counterparts. 

Tester: Flumo and Yakpalo always drink canejuice (rum) together. Flumo is 
drinking cane juice. Is Yakpalo drinking cane juice? 

Subject: Flumo and Yakpalo drink cane juice together, but this time Flumo 
was drinkingcthe first one. Yakpalo was not there on that day. 

Tester: ' But I told you that Flumo and Yakpalo always drink cane juice together. 

One day Flumo was drinking cane juice. Was Yakpalo drinking.cane ^ 
juice that day? 

Subject: The day Flumo was drinking the cane juice Yakpalo was not there 
on that day. 

' Tester: What is the reason? 

Subject: The reason is that Yakpalo went to his farm on that day and Flumo 
remained in town that day.^ 

The tester reported that his subjects often responded by saying something like "Yakpalo isn't here 
at the momerrt^ why don't you go and ask him about the matter?" 

These exchanras show that the Kpelle are a clever people. The problem here is that the 
subjects do not accept the ground rule that is automatic with academic people: Base your answer 
on the terms defin-Bd by the teacher (or tester). 



Cole, M., and Scnbner, S. Culwreand thought. New York: Wiley, 1974. 



In school, a problem should be tackled in the same way no matter 
' which teacher assigns it to you (teachers keep chsujging); numerical problems 
^ . t ^reworkedoutslmilarly regardless of whether apples or bombs are to be 

added -(or without any specific numbers at all, as in algebra); geography is to 
• be mastered whether or not one has any interest in traveling. The school child 
learns to use one particular skid or heuristic to solve m. different puzzles, 
even If they diffeR in any details, simply because they share an abstract struc- 
ture. He also learns to w6rk on problems as th^y are presented, whether he 
cares aboat the solutions or not. It is the absence of these attitudes among 
the unscho^oled Kpelle thgit accounts in large part for the sorts of answers 
theygive tb questions. 

School shapes our thinking in other ways. 

We are expetfted to leave our life situations at the door, as it were, and 
to solve problems that other people have set. Notice also that problems on 
school, tests are supposed to be "f§ir"-that is, all the information needed to 
solve them is typically given from the beginning. The pupil does not find out 
anything as he goes along that, might have been otherwise. I will call problems 
of this kmd puzzles," because they are so different from the problems of 
^ ordinary human life. To solve "puzzles" like this requires particular skills, 
often skills of a high order. It is appropriate tp call them academic skills.® 

Academic skills and puzzles are challenging but mdy be useless and constraining ou 
school siting. 



In general, assessment oi transfer skills is the evaluation of the capacity of the learner to 
adapt to new work and life situations, not to solve "puzzles." The assessment of the abilities 
previously mentioned in the context of occupational adaptabflity is quftte different than assf ssment 
within the context of other areaspf education and psychology. For exam.ple, the purpose of«thr 
assessment of the level of a high school student's ability to reason by analogy is. usually for college 
admissions. The assessment of a very young student's ability to reason by analogy. is usually for the 
purpose of determining whether the child has an acceptable level of mental capacity for functioning 
in the normal public sichool setting. These are assessments related to academic skills. The assessment 
of the ability of the learner to transfer one or several skills from one occupational setting to another 
might have quite a different purpose. True, assessment, among other things, could inform the 
educator or employer about the presence or absence or level of mastery of various abilities that, for 
example, assist the process of transferring skills acquired .by a homemaker in planning family meals 
to a job as hospital dietician. However, it is assumed that, rarely, if ever, would her "score" (if a 
norm -referenced score was possible) on any measure of transfer ability keep her from obtaining 
entry to that position. It is what she can do or the observed behavior which shows transfer has 
occurred'that is essential. 



Neisser, U. Academic and artificial intelligence. In L. B. Resnick (Ed.), The nature of intelligence. New York John Wiley & 
Sons, 1976. 



What is Meant by Assessment? 
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• Assessment in the context of the transfer process should contain elements of learning and 
diagnosis. Assessment should not only be concerned with monitoring and evaluatmg a learner s 
transfer facility, but should actually promote this facility. Unlike the testing situation of the 
Kpelle's above, which leads to a "score" that serves to label an individual or group, assessment V. 
should be a primary aspect of the educational process itself and. in fact, should be integrated with 
such matters. 

The following should be general characteristics of the assessment sittiation: 

1. Multimodal in presentation and response {not just tests consisting of multiple-choice 
items) 

2. Related to stated outcomes of training , 

3. Relevance or similarity to real-life situation 

4. Practice and warm-up ^ 

5. Feedback that is prescriptive or diagnostic 
*6. Ongoing and continuous 

V 

Finally. a«essment here should be clinical rather than statistical, individual rather than group. 
It should be as informative to the learner as it is to the teacher. 

How Can Transfer Skills Be Assessed? 

There are two types of educational measurement that can be applied to the assessment of 
transfer skills as defined above. Traditional assessment tools are objective, multiple-choice tests. 
These tests can be scored numerically and scores are usually referenced to a normative group. Non- 
traditional methods are usually those requiring open-ended responses, products, or performances. 
They are very often characterized by having a real-world flavor. These nonobjective responses are 
scored by expert judges who use guidelines and criteria for making such judgments. The information 
below is intended to give the reader an overall picture of the two methods including their capabilities 
and limitations. ^ / 

Objective Tests, there are a number of published instruments that have been designed to 
measure the abilities listed previously. They are paper-and-pencil devices which contain, in the mam. 
multiple-choice items. Their advantages are that they are economical and permit an efficient and 
objective assessment of various skills and abilities that might be needed for transfer. However, there 
are characteristics associated with multiple-choice tests that are at odds with the characteristics of 
the assessment situations suggested in the previous faction. They tend to b6 artificial, resulting in a 
lack of face validity. Learners may not see the connection between the test and what is claimed to 
be measured. The tests are unimodal in presentation and response. A considerable amount of 
reading is required and therefore the level of skills that are being measured are confounded with the 
learner's level of verbal aptitude, when verbal skill may not be correlated with the skill being 
assessed. In using these tests, one runs the risk of testing for "school-based" or academic skills. 
However, these instruments are convenient and can be useful for practice or warm-up. The resulting 
scores can be informative when used in conjunction with more real-world, nontraditional assessment 
methods. Published test exist that measure one's cognitive ability, factor, or skill; others contain 
sections or subSsts that measure cognitive abilities that may be associated with the transfer pro- 
cess. However, further investigation and analysis of the content of such tests is needed, in order 
to support the validity of the relationship betwee.i test outcomes and transfer skill acquisition. 
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Care must be taken in the selection and use of any tests, especially when assessing transfer 
ability. Such tests are to be used mainly for guidance, diagnosis, feedback, or evaluation rather 
than for placement or promotion. For example, a manual associated with a test of synthesizing 
ability may indicate that extensive studies have been done to show how well the test predicts 
academic performance (predictive validity). This should be noted, but other information may be 
more important for teaching for transfer. Does the test look as if it measures what it purports to 
measure? What is the reading level of the items? Is it self-scoring and does it provide feedback to 
the learner? Do the items relate to real-world behaviors? 

Reliability of the test would be important for any situation in which it might be used. The 
reliability of the instrument indicates the amount of error present in any score or rating for an 
^ability and basically shows the degree of consistency of measurement when using a particular 
procedure or test. Learners should be assured that even though the test is mainly to provide practice 
in transfer and guidance, it is fairly reliable across' time and situations. 

Practical and substantiv^ questions to consider when evaluating and selecting a published test 
for use include: * 

1 . What is the purpose and use of the test and how does it relate to the purpose of your 
training and assessment? 

2. How and why was the test developed? 

3. What sorts of data and norms have been collected in the course of test development? 

4. Do the data specify age, sex, educational background, socioeconomic status, and 
geographic distribution of the subjects used for test development? 

5. Are there any sex or cultural biases refle'jted in the test? 

6. Is there any unethical practice associated with the test and is there truth-in-assessment? 

7. How are scores reported and what information is available to assist in interpretation? 

8. Is it easily available or is it so secure that it is difficult to purchase? 

9. What type of training is necessary to administer the test? 

10. How long does the test take? 

11. What are all the costs associated with the test? 

12. What are the scoring options associated with the test? Self sco. ing? Scored by the 
test publisher? 

Nontraditional Methods.' Objective tests are the most practical and efficient, and thus most 
traditional, method for assessing cognitive skills associated with skill transfer. If, however, transfer, 
skill assessment is to have educative benefit and contain elements of realism that are essential to facp 
validity, other less frequently used-thus nontradit.^ lal -assessment procedures may be indicated. 
These procedures have the added advantage of usually providing more immediate and personalized 
fee^jback to the learner. Unlike multiple-choice, objective examinations, these procedures use expert 
judgment rather than scoring that is based on norms and statistical studies. Experts usually rate the 
learner on cognitive behaviors or dimensions that show successful transfer using a variety of materials. 
Then one or several persons must decide, through consensus, on the extent and level of learning. It 
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should be noted that several measurement techniques can be used tb assess whether transfer has 
occurred. This Is especially true when a great deal of error is associated with each technique or 
when the measurement procedures are far removed from direct observation of student performance. 
Some measurement techniques which have been developed in the fields of psychology, business, 
industry, and the military will be described briefly. Appendix A is an annotated bibliography of a 
selection of documents that explain some specific assessment methods more fully. 



Interviews and Oral Examinations. The assessment interview is a face-to-face interpersonal 
situation in which the interviewer asks the candidate questions designed to obtain answers which 
will lead to an evaluation of transfer ability. During the interview, skills, knowledge, or concepts 
attained through experience can be listed, described, and later evaluated; and a simulated problem 
or situation may be presented to determine whether transfer has occurred. The interview can be 
used to "talk out" solutions to problems. 

In general, questions in the interview should be: 

1. directly related to the skills and knowledge the assessor wishes to assess, 

2. appropriate in type for the kinds of information to be obtained, 

3. clear and unambiguous, 

4. free of leading questions, 

5. requesting only that knowledge or information that the learner is expected to have, 

6. free of questions that require personal information that the interviewee may resist 
revealing, 

7. constructed so that the interviewee does not offer the socially acceptable response. 

The interview as a measurement technique is adaptable to individual situations and, in most 
cases, allows for probing in order to obtain more ihformation. The major disadvantage of the inter- 
view is that it takes much more time and tends to be less valid and reliable than more objective 
techniques. 

In the standardized or structured interview, the questions, their sequence, and the wording are 
fixed. The interviewer can take very little liberty in changing the interview format. The highly 
structured interview can be administered by tape, in person, or in a written form. Hov»^ever, the 
responses to such an instrument are oral. The unstructured interview is much more flexible and 
open. Topics, issues, and questions are outlined, but the content of the questions, their sequence, 
and the wording are entirely up to the interviewer. In reality, the assessment intei view is a combi- 
nation of both techniques. 

An effective assessment use of a semi-structured interview would be to present orally to a 
learner possessing a set of transferable skills a problem and situation quite different from that in 
which these skills were learned or last practiced. After presentation and discussion, the learner would 
be asked questions about application of these skills to the new setting. 



Simulations. In simulations, sometimes called situational tests, the learner is asked to pretend 
that he or she is engaged in some realistic task, the nature and content of which are described in 
some detail before his or her role is assumed. Often the simulation is less complex and more con- 
venient than the real situation and operates under a compressed time schedule. Such a procedure 
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offers the assessor and learner the unique opportunity to observe and measure the quality of 
performance in a life-like setting that often cannot be measured by other means. It also offers the 
learner an opportunity to practice transfer skills or be trained in transfer. Simulations are ordinarily 
used to assess complex qualities such as analytical thinking, goal setting, risk taking, interpersonal 
competenc-j, decision making, sensitivity to the behavior of others, oral communication, and planning 
skills, many'of which influence the transfer process. However, a major limitation is the more faithful, 
comprehensive, and thus valid, the exercise, the more costly the procedure. 

For situatioHu! tests or simulations to be of maximum assessment and educational value, the 
follovwing rules should be cbserved: 

1. Specific behaviors relevant to the variables being measured should be identified before 
the setting and problem situations are developed. 

2. Problem situations should be carefully structured to provide the opportunity for 
candidates to act in a satisfactory or unsatisfactory manner with respect to the specific 
behaviors. 

3. The amount of fidelity to rsality for each aspect of the simulation must be determined 
and weighed against the limits of available resources. 

4. I nstructions must be sufficiently clear so that the candidate knows what is expected. 

b. Tasks included in the exercise should not require knowledge that the candidate is not 
ordinarily expected to have. 

6. Standard rating instructions and multiple judgments are needed to achieve validity. 

7. The designer of an exercise must have a thorough knowledge of the experience being 
simulated. 

Simulation technology has generated a number of specific assessment devices which can also be 
used for training, some of which will be described below. 

Educational games as simulation exercises can be categorized as (1 ) media-ascendant simulation 
games which are mediated by machines; (2) interpersonal ascendant simulation games, in which 
decision making, role playing, and player interaction are emphasized; and (3) nonsimulation games, 
which merely provide a competitive context for learning and practicing concepts and principles. 
Academic games are used in the assessment of problem-solving and decision-making skills. 

Academic or management games could be one of the most economical assessment techniques 
in that virtually hundreds of them are available through companies that specialize in manufacturing 
educational products. Since the scoring and rating behavior during an educational game is usually 
complex and therefore subject to error, the games shouldn't be used as the sole measurement of 
problem-solving or decision-making ability. 

Some caution should be exercised when using games for assessment purposes. Little research 
or discussion has been focused on measurement problems concerning the use of qames for assess- 
ment purposes. Several factors inherent in the technique might have a tendency to confound out 
comes and thus interfere with effective assessment. For example, performance in gaming can be 
influenced by variables such as tolerance for ambiguity and risk taking. These in turn are moderated 
by sex, minority group membership, and socioeconoir.ic class, some of the same factors that influence 
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the transfer process. If these variables influence problem-solving behavior and decision making in 
the world of work^ the use of games to measure these skills is justified. However, if factors such as 
risk taking only influence behavior in the gaming situation, this sample of the candidate's behavioi 
would not be an accurate representation. Also some individuals who participate in a game as an 
assessment technique may resent being judged by winning or losing. Despite these problems, games 
could be powerful training devices. 

When the number of learners is large, the expense of simulation is too great, and/or the 
assessor is interested in obtaining data from the candidate on specific crucial information and skills 
required to solve a problem or diagnose a malfunction (human or machine), the written simulation 
is appropriate. It is a paper-and-pencil situation which may follow any number of written formats, 
even mulrlple-chotce or essay. 

The case study simulation involves a f3irly complete and realistic statement of some problem 
that can be related to business, human services, or almost any area of human experience. The candi- 
date is asked to write an'analysis of the problem in a specified period of time, and his/her report 
should contain alternative solutions and recomrfh^ndations for action. The exercise is used to assess 
the candidate's ability to analyze and solve compliex problems. Sometimes the case under considera- 
tion is not written. It can be presented to the candidate in several modes, such as f ilmstrip, audio- 
tape, or videotape. 

The in-basket exercise takes its name simply from the in basket or tray found in most 
businesses and institutions where employees work from an office or desk. Forms of this exercise 
are being used to assess police officers, military officers, and school principals. The exercise simulates 
a specific organizational role, requires the examinee to deal with problems as if he/she were actually 
on the job, and provides a set of materials which may contain incoming letters, memos, reports, 
papers to be signed, and telephone messages. 

All student decisions and actions are recorded in writing ar.d may include letters, memos, 
outlines, directions, and the content of phone calls. These written products are usually scored by 
trained evaluators using methods sin.ilar to those used for essay grading. More efficient scoring 
techniques have been devised wherein a checklist is provided to the candidate, and he/she picks out 
the actions performed from a wide list of possible actions. The learner may also add any unusual or 
unlisted actions taken. These resp^ ises are judged a id scored by trained assessors or the learner. 
At times, an interview after the exercise is used in lieu of scoring procedures. 



Essays. Essay examinations are instruments which are characterized by the presentation of 
questions or tasks to which the student responds by organizing and writing an ansvier. Some tests 
are based on the student giving written responses to situations presented on audip-^or videotape. 
Usually, no single response or pattern of responses can be listed as correct in th^f^ven an expert 
cannot usually classify a response as categorically right or wrong. Rather, there are different degrees 
of quality or merit which can be recognized. Essay content may be factual or creative. 

The following steps should be considered in constructing essay assessment instruments: 

1. Define »^bjectives clearly so that tasks may be constructed which are related to the 
framew(,rk of objec'jves. 

2. Formulate weM-focused questions and define the task through determinir^g the amount 
of freedom oT response. 

# 
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3. Construct several model answers while taking note of writing time. 

4. As a pretest check, have an examiner who didn't formulate the questions answer them. 

5. Ask colleagues or specialists to review the questions. 

6. Pretest the items so that the following can be answered: 

a. Do students appear to understand the intent of the question, or do they appear to 
interpret it in ways not intended? 

b. Is the question of appropriate difficulty for the examinee population? 

c. Is it possible to grade answers reliably? 

7. Recognize the importance of an adequate sample of questions (a sufficient number, the 
right number of short and long questions, and alternate questions from which to choose). 

The main source of measurement error when essays are used for assessment is associated with 
the rating of essays. Essay rater research has shown that: 

1 . Different raters tend to assign different judgments to the same paper. 

2. A single rater tends to assign different judgments to the same paper on different 
occasions. 

3. The differences tend to increase as the essay question permits greater freedom of 
response. 

These findings ai e probably direct! related to the values inherent in the educational situation 
as well as the values of the raters. If es'- s are "academically loaded." then they would not differ 
in na'^ure from most multiple-choice ex ...ninations. If the essay questions do not mere y test academic 
skills thev could be used in assessment situations which include demonstrating the following t-ansfer 
skills: information access and production, short- and long-term planning, value analysis, synthesis, 
and critical thinking. ^. 

1 

Performance tests. There is a stronger element of rer.lism in performance tests than in simula- 
tions because they rfisemble the situation in which a specific kind of experiential learning is applied 
more closely. Frequently, a performance test is nothing but a work-sample which requires the 
accomplishment of specific tasks in a controlled setting. Or it may consist of a situational observa- 
tion of performance in a natural setting. These procedures are not as flexible as simulations since 
they don't always allow for changes in setting or context. In assessing transfer skills, it is suggested 
that the setting or situational variables be changed. 

A work sample attempts to reproduce all or an important part of the actual operations and 
tasks of the job, frequently utilizing the actual equipment used on the job. Sometimes the test 
Involves certain difficulties typical of the job which the examinee must overcome. Some of the 
realism of the work setting may be sacrificed, but the critical job elements are present and the test 
is readily recognized as a realistic representation of the tasks one would etvsounter on the job. 

Interviews are suggested to follow up the performance test. In this way, the teacher or assessor 
can determine whether the learner is mimicking behavior or really understands how his or her 
transferable skills are being transferred. 
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Self-astes»m«nt. Self -assessment or self-evaluation can be regarded in general as a set of 
relatively undefined techniques by which the student judges his/her own level ot accomplishment. 
Since the student is likely to be knowledgeable about past accomplishments, it is rcas*^ .'ole to 
assume that with appropriate guidance and criteria he/she pan provide accurate indication of his/her 
present abilities. Self-evaluation provides the feedback to the ledrner that is so necessary for acquiring 
educative benefits from assessment. It has been shown that the transfer process is more rapid if the 
learner can discover relationships for him- or herself. 

Self-assessment can be an integral part of all the measurement techniques described above. 
Students, with the help of explicit criteria and rating forms, can rate themselves on their performance 
in such measurement situations as simulation exercises, performance tests, or essay examinations. 
The learner's fatings and judgment will reflect his/her perceptions about the extent and level of 
specific learning transfer. 

The ratings can be compared to the ratings of assessors resulting in an evaluation conference 
with the student's teacher, assessor, and/or counselor. Discrepancies can be resolved; areas of strength 
and weakness can be pointed out. Also, it may be that the learner will be asked to do some supple- 
mentary or practice work in areas where this extra effort will boost the level of transfer. In addition 
to self-rating, self-assessment could include procedures in which the learner "talks out" solutions to 
the problems presented during assessment so that the learner can learn more about his or her 
problem-solving ability and style. 

Although the informal methods of self-assessment (for example, periodic discussions between 
a student and teacher) have the appeal of flexibility and individuality, their subjective nature and 
lack of uniformity make them difficult to evaluate and address without training and practice. In 
addition to its assessment function, realistic self-appraisal is useful both for the identification and 
improvement of weak transfer skills and for motivating transfer of Jearning. To a large degree, the 
accuracy and the utility of the self-assessment process will depend on the effort that student and 
teacher expend on it and the amount of guidance or training that the student receives in the process. 



Conclusion 

The techniques described in the previous section cannot be prescriptions for measuring the 
extent and level of transfer skills at this stage in the investigation of the transfer process. It would 
be naive and premature to think any specific tests or procedures could be recommended. The array 
described in this paper is meant to show learners, teachers, and program evaluators that there are 
aids available to them. These tool? can help them determine whether transfer has occurred and 
serve them in the educational process. At present, there is no manual or set of procedures that will 
inform these individuals as to how these tools can be melded with transfer process. Such operatioris 
are left to those creative minds in research and education that are challenged by the notion of testing 
and teaching for transfer through assessment. 
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Bibliography of Selected Documents 
Describing Nontraditional Assessment Methods 



Interviews and Oral Examinations 

How to Interview by W. Bingham and B. Moore. New York: Harper, 1959. 

A simply written guide to constructing and performing the interview, this book contains an 
entire chapter on oral examinations. 

"How Reliable are Good Oral Examinations?" by H. D. Carter. California Journal of Education, 
1972, 13, 147-153. , 

To ascertain the reliability of the oral exam, the author analyzed data from a sample of 250 
candidates examined in a field of medical specialization. The results indicate that the oral 
exam is highly reliable and that there is a low relationship between the oral exam and a written 
exam covering the same subject area. 

The Selection Process: Choosing the Right Man for the Job by M. M. Mandell. New York: The 
American Management Association, Inc., 1964. 

Although this book emphasizes techniques concerning the hiring of individuals in eight 
different occupational groups. Chapters 12, 13, and 14 are particularly informative on the 
subject of interviewing, and much of the material can be interpreted in the light of assessment 
rather than selection. The chapters contain descriptions of the stress inten/iew and the group 
oral examination. 

"On Maximizing the Information Obtained from Science Examinations-Written and Oral" by 
J. E. Piatt. American Journal of Physics, 1961,25, 111-122. 

The author regards the written examination as a mapping function in which the student's 
abilities are mapped on a numerical scale. An oral exam is described as a search path to locate 
the boundaries of a student's knowledge. Common pitfalls of the oral exam are described. 



Simulations 

Serious Games by C. C. Abt. New York: The Viking Press, 1970. 

This book explores the ways in which games can be used to instruct and inform. Chapters 
examine topics such as evaluating the cost-effectiveness of games and improving education 
with them. The use of games in general areas such as the physical and social sciences, vocational 
guidance, and government and industrial planning is discussed. 
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A Primer on Simulation and Gaming by/ R. B. Barton. Englewood Cliffs, NJ.: Prentice-Hall, 1970. 

This book combines computer and noncomputer approaches to simulations and gaming but 
does not require learning a computer language. A checklist for conducting man-only and man- 
computer simulations is of practical value. 

"Issues in the Development and Validation of In Basket Exercises for Specific Objectives" by 

L A. Crooks. Research Memorandum (RM-68-23), Princeton, N.J.: Educational Testing Service, 1968. 

This paper discusses the practical issues surrounding the development and use of the in-basket 
test. 

"Why Do We Use Situational Performance Tests?" by D. W. Fiske. Personnel Psychology, 1954, 7, 
4r4-469. 

The article discusses sources of error In measurement when using situational tests. 

"Some Considerations in the Development of Situational Tests" by J. C. Flanagan. Personnel 
Psychology, 1954, 7, 461-464. 

The author explains the advantages and disadvantages of using situational tests to measure 
attitude, motivation, and other personality factors. 

"Factors in In-Basket Performance" by N. Frederlksen. Psychological Monographs, 1962, 76, 
(Whole No. 541 ). 

A factor^analysls of the 70 aspects of behavior used to rate 335 people on an in-basket test 
revealed 8 primary factors, including informality, directing subordinates, discussing, and 
acting in compliance with suggestions. 

"Do Filmed Cases Improve the Case Method?" by T. B. Green. Training and Development Journal, 
1973,27,28-31. 

The article sum*fnarlz3S a study in which the case method was combined with a film approach. 

"A Management Tool: The In-Basket Exercise" by F. M. Lopez. Simulation and Gaming News, 
1973, /, 1 and IS. 

The article describes the Quasar In-Basket Exercise which can be adapted for use in almost 
any organization. It includes a method of scoring via the^ Action Report, either by mechanical 
or electronic means. 

"Simulation Technique In the Measurement of Problem-Solving Skills" by C. H. McGuireand 
D. Babbot. Journal of Educational Measurement, 1967,4, 1-9. 

This article r«e«:ribes the development of a series of branched problems in patient manage- 
ment that require sequential analysis and decision. The problems we«-e designed to measure 
aspects of behavior defined by a criterion group as essential components of clinical competence. 

"The Name of the Game is Simulation" by S. B. Parry. Training and Development Journal, 1971, 
25, 28-32. 

The writer explains the rationale for using games and simulations, with some examples, and 
makes the Interesting distinction between simulations designed as application of prior learning 
and those intended to give the learner an opportunity to gain insight into his own behavior. 
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"Critical Incidents in Counseling: Simulated Video Experiences for Training Counselors" by 
J. D. Spivack. Counselor Bducation and Supervision, 1973, 12, 263-270. 

This article -explains the rationale, development, and use of a simulation approach to 
training counselors in which videotaped vignettes of critical incidents that occur in the 
course of establishing and maintaining a helping relationship are presented to tramees as 
stimuli. ^ 

"Sources in Simulation and Academic Gaming: An Annotated Bibliography" by P. J. Tansey and 
D. Unwin. British Journal of Educational Studies, 1969, //, 193-208. 

This bibliography attempts to reduce more than 2,000 articles on simulation to 87 citations. 
The articles selected concentrate in the fields of school and college learning with particular 
emph.'isis on simulation as a teacher training tool. Some articles on role playing and sociodrama 
are included. .1^^, 

"Assessing Clinical Judgment" by J. W. Williamson. Journal of Medical Education, 1965, 40, 180-187. 

The author describes methods of analysis for data generated from branched problems in 
patient management. 

Essays 

"E«5say Examinations" by W. E. Coffman. In Educational Measurement, R. L. Thorndike, Ed. 
Washington, D.C.: American Council on Education, 1971, 271-302. 

This chapter gives a systematio analysis of essay examinations and describes their construction, 
limitations, scoring procedures, technical problems, and needed research in the area. 

Measuring Educational Achievement by R. L. Ebel. Englewood Cliffs. N.J.: Prentice-Hall, 1965, 
84-123. 

Chapter 4,' "The Characteristics and Uses of Essay Tests," is of value to the classroom teacher. 
Special consideration Is given to guidelines for constructing teacher-made essay tests. 

"Schools of Thought in Judging Excellence of English Themes" by J. W. French. In Proceedings, 
Invitational Conference on Testing Problems, 1961. Princeton, N.J.: Educational Testmg Service, 
1962, 19-28. 

This paper investigates the reasons why essay graders disagree and what information the essay 
response, when properly graded, can offer that the multiple-choice response cannot, 



Performance Tests 

Handbook on Performance Testing, A Practical Guide for Test Makers by J. L. Boyd and B. Shimberg. 
Princeton, N.J.: Educational Testing Service, 1970. 

This book describes how to plan, develop, and grade work sample tests, and the types of docu- 
ments needed for their administration. It provides a portfolio of tests, such as those for 
machinist, cosmetologist, dental hygienist, and bench woodworker. 
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"Performance and Product Evaluation" by R. Fitzpatrick and E. J. Morrison. In Educational 
Measurement {2nded.), R. L Thorndike. Ed. Washington, D.C.: American Council on Education, 
1971,237-270. 

This chapter describes work sample tests and distinguishes them from other kinds of 
simulated testing situations. 
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REPORTS ON OCCUPATIONALLY TRANSFERABLE SKILLS 

McKinlay. B. Characteristics of jobs that are considered common. Reviem of literature and research (Info. Series 
No. 102). 1976. ($3.80) 

A r »view of wious approaches for classifying or clustering jobs, and their'use in (a) describing the elements 

• of oommona..ty involv'ed when people make career changes, and (b) understanding better the concepts of 

• occupational adaptability and skill transfer. 

Altman. J.W. Transferability of vocational skills: Reviev^ of literature and research (Info. Series No. 103). 1976. 
($3.80) 

A review of what is known about the transferability of occupational skills, describing the process or the 
r facilitators of skill transfer. 

Sjogren. D. Occupationally transferable skills and characteristics: Review of literatyre'and research (Info. Series 
No. 105). 1977. (S2.8U) 

A review of what is known about the range of occupation-related skills and characteristics that could be 
considered transferable from one occupation to another, describing those transferable skills which are 
teachalple.in secondary and postsecondary career preparation programs. 

Ashley. W.L. Occupational information resources: A catalog of data bases and classification schemes (Info. Series 
. No. 104). 1977. (S18.20) 
Aquick and concise reference to the content of 55 existing occupational data bases and 24 job classificat-on 
schemes. Abstracts of each data base and classification scheme include such information a^;. 'dent' .>« -on 
investigator, location, documentation, access, design information, subject variables, occupation variables, and 
organization variables. 

Wiant. A.A. Transferable skills: The employer's viewpoint {\nio. Series No. 126). 1977. ($3.25) 

A reoort of the views expressed in nine meetings across the country by groups of local community and 
' bus!ne" representatives concerning the types of transferable skills required and useful in their ^o^^ 
and how s better understanding of transferable skills could improve training and occupational adaptability. 

Miguel. R.J. Developing skills for occupational transferability: Insights gained from selected programs (Info. Series 
No. 125). 1977. ($3.80) 

A report of clues and suggestions gained .n the review of 14 existing training prograrris. with recommendations 
Tor practice which appear to have been successful in recognizing skill transfer and taking advantage of an 
mdividuars prior skills and experience. 
Ashley. W.L.. & Ammerman. H.L. Identifying transferable skills: A task classification approach {R&D Series No. 
146} 1977. 

A report of an exploratory study designed to test the usefulness of three classification schemes ... identifying 
the transferable characteristics of tasks in diverse occupations. 

Pratzner, F.C. Occupational adaptability and transferable skills (Info. Series No. 129). 1977. ($6.25) 

A summary fmal report, presenting and d-scussmg an ariay of issues encountered in the various project 
activities, and offering recomrriendations. 

Selz N.A.. & Ashley. W.L. Teaching for transfer A perspective for practitioners (Info. Series No. 141). 1978. (82.35) 
' An informal discussion of the need for teachers and trainers to give more attention to developing transfer- 
fb" ^and ransfe^lw^^ m students for learning and life performance applications. Practical suggestions 
:nd techntq "e^f or "rnprovin the capacity of students to transfer learned skills anci knowledge to new 
Situations are given. 
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Brickell. H.M.. & Paul. R .H . Minimum competencies and transferable skills: What can be learned from the two 
niovements (Info. Series No. 142). 1978. ($5.10) 

A report comparing and contrasting potential impact of the transferable skills and minimum competency 
testing movements on school programs, staff, and students. Key questions and alternative strategies are 
presented to assist educational planners and administrators in formulating policy and establishmg prortJotion 
or completion criteria in secondary and postsecondary education. . 

THE FOLLOWING REPORTS WILL BE AVAILABLE IN 1980: 

Ashley. W.L.. Laitman-Ashley. N.M.. and Faddis. C.R. (Eds.) Occupational adaptability: Perspectives on tomorrow's 
careers ( I nfo. Series No. 1 89) . 1 979. 

Proceedings from a national symposium. The topics focused on how training for adap ability can increase the 
use of human resources in the labor force. 

Selz. N . ( Ed .) Adult learning: Implications for research and policy in the eigh ties, 1 979. 

Proceedings from a national symposium on adult learning. Topics include state of the art. research into 
practice, policy implementation, and future directions. 

Wiant. A.A. Self-assessment for career change: Does it really work? Summary report of a follow-up study 
(Info. Series No. 191). 1979. 
• An analysis of the impact of self-assessment on one's subsequent employment experience. The particular 
assessment technique studied is one intended to help identify those skill attributes which have provided 
satisfaction in varioi|s life experiences. Outcome measures included skill utilization and job satisfaction. 

Selz. N.A.. and Jones. J.S. Functional competencies in occupational adaptability and consumer economics, 1979. 
' Perceptions ttf national adult samples are reported. Document includes where competencies should be taught 
-at home, at school, on-thetjob. self-taught-and how important these competencies are in successful work 
and life activities. 

Kirby. P. Cognitive Style, learning style, and transfer skill acquisition, 1979. 

A review and synthesis of the literature in adult learning styles, as they relate to the acquisition of transfer 
skills. 

Knapp. J.E. Assessing transfer skills, 1979. 

A review of traditional and non-traditional assessment with respect to the assessment of transfer skills. 

Sommers. D. Empirical evidence on occupational mobility (Info. Series No. 193), 1979. 

A review and synthesis of- the literature on the characteristics of occupationally mobile workers and their jobs. 

Lattman-Ashley. N.M. (Ed.) Women and work: Paths to power (Info. Series No. 190), 1979. 

Proceedings from a national symposium that offer perspectives on women in the work force. Topics will cover 
five major transition points that any person can experience in a lifetime. 

ORDERING INFORMATION 

All prices include postage and handling. When ordering use series numbers and titles. Order of SIO.OO ot less will 
be accepted on a cash check, or money order basis only. Purchase orders will be accepted for orders in excess of 
$10.00. Please make check or money order payable to: THE NATIONAL CENTER FOR RESEARCH IN 
VOCATIONAL EDUCATION. Mail remittance and/or purchase order to National Ce ter Publications, The Ohio 
State University. 1960 Kenny Road. Columbus. Ohio 43210. (Prices subject to change.) 
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